The scene presented to the pilot in a flight simulator is generated by a video projector projecting spatial registered red, green and blue component images onto a screen as controlled by information stored in a computer memory. To reduce the volume of digital video data that has to be stored in the computer memory and that has to be previously processed in the flight simulator computer, color information is preferably expressed in Y,I,Q color coordinates. In the Y,I,Q color coordinates defined by the National Television Standards Committee (NTSC), Y is a luminance component, I is the color saturation component extending along the orange-cyan axis, and Q is the other color saturation component. In the flight simulator the I and Q color component samples are subsampled respective to the Y color component samples in the vertical direction of image scan, as well as the horizontal direction, in order to reduce further the number of digital samples that need to be processed in order to describe a field of view with given luminance (Y) resolution. A 4:1 subsampling of I and Q respective to Y in both horizontal and vertical directions is employed in flight simulators marketed by General Electric Company. The use of Y,I,Q color coordinates and a further transparency (T) coordinate in image descriptions provides for the simplified generation of montage images by the flight simulator, as known in the art.
Polygonal models of trees, runways, buildings, etc. are stored in the memory of the flight simulator computer, to be used in generating the video simulations of take off and landing. In more recent flight simulators textures are mapped onto the surfaces of the polygonal models by video superposition methods to give the models more realistic appearances, and these textures may be generated proceeding from images created by the scanning of these textures or photographs of them with a video camera and digitizing the video camera video signal with an analog-to-digital converter.
A polygonal model of the terrain to be overflown in the simulation is also stored in computer memory, as well as a terrain map to be texture mapped onto the polygonal model of the terrain, to generate digital samples descriptive of the pixels in the video simulation of the underlying terrain as viewed during flight. The terrain map can be generated as a composite image proceeding from a number of overlapping images, each created by scanning the aerial photographs with a video camera and then digitizing the video camera video signals with analog-to-digital conversion apparatus.
Considerable processing must be done to the individual images before splining them together to form a composite image. Gamma corrections should be made for the aerial camera film, for the video camera, and for the analog-to-digital conversion apparatus. Gamma correction is preferably made at each point in the procedure just after a gamma error has been incurred, but gamma corrections may also be made on the digitized samples generated by the analog-to-digital conversion apparatus. The distortion effects owing to camera angle and perspective are corrected for, and optical distortions in the camera lens may be corrected for. Optical color corrections as between images may be done where called for.
The intensity differences between overlapping images must be reduced in the overlap regions to implement splining. There are several mechanisms that can give rise to intensity differences between overlapping images. Illuminant changes may occur from exposure to exposure owing to overhead clouds or to intervening haze, smog or smoke. The camera may be provide automatic exposure time control that attempts to keep a photograph within the dynamic range of the film; such systems usually have center-weighted light metering which largely ignores the periphery of the image.
An important optical image plane effect is known as the cos.sup.4 law. A decreasing of image irradiance that depends upon the "field" angle between an image point and the principal point, where the optical axis meets the image plane, occurs which decrease follows the fourth power of the field angle in a simple lens in which no vignetting takes place. The effect affects compound lenses as well, though the decreasing of image irradiance as a function of the "field" angle may stray from the cos.sup.4 relationship. In wide angle lenses such as those used in aerial photography, the loss of illumination due to the cos.sup.4 law is appreciable.
"Vignetting" is the restriction, by multiple apertures, of light passing through an optical system. Image locations within a certain angle from the optical axis (which depends upon the optical system) are not affected by vignetting. The amount of light which reaches image locations outside that angular cone falls off rapidly. For complex imaging systems, vignetting can be an important problem. Interestingly, "anti-vignetting filters" are used to correct for the cos.sup.4 falloff, rather than vignetting, which is f-number dependent. The filters, which are darker in the center, gradually becoming more transparent toward the perimeter, are usually designed to correct less than the full cos.sup.4 falloff, to avoid dimming the image too much.
In aerial photographs analyzed by the inventors, however, image intensity losses are observed that are attributable to effects beside cos.sup.4 law. While the causes of these image intensity losses are not understood very well, another prime suspect is the effect of terrain angle, relative to the camera and the sun. The Malibu, Calif., region in the aerial photographs analyzed by the inventors has a sharply inclined terrain, which may contribute to the apparent variation in illumination.
A plot of the frequency with which each image intensity level occurs in an image or portion thereof, as plotted against the image intensity level scale, is referred to as the "histogram of image intensity level occurance in that image or portion thereof" and is referred to herein in abbreviated form as simply the "histogram" of that image or portion thereof. A standard method for reducing intensity differences between overlapping images involves finding a function which maps the histogram of one image to that of the other image in the overlap region they share. That mapping is then applied to the entire image, equalizing the histograms of the two images in the overlap region and normalizing the intensities throughout. However, as simply applied, this method relies upon there being little or no radiometric distortion in the images--i.e., that the intensity difference between corresponding elements of the images is substantially independent of the position of those elements in their respective images. The method, as simply applied, does not work very well in the presence of position-dependent intensity variations.
Several digital image processing methods exist for calibrating sensors to correct for radiometric distortion. These methods involve processing standard inputs with the optical system to be corrected. However, it may not be possible to obtain calibration data, especially after the images have already been obtained.
The inventors have observed that the position-dependent intensity variations tend to be a lower-spatial-frequency phenomenon. Satellite photographs provide terrain overviews that are generally considered to have insufficient resolution to generate sufficiently detailed terrain maps for a flight simulator to be able to satisfactorily simulate flight at lower altitudes, but these satellite overviews can span large areas. Over a region depicted by overlapping aerial photographs, such a satellite overview can correspond to the lower-spatial-frequency content of the aerial photographs. It is postulated that satellite data (or other very high altitude data) for the same region have more constant intensities for those lower spatial frequencies. The method of the invention discards the lower-spatial-frequency content of a mosaic of the aerial photographs, rather than attempting to correct that lower-spatial-frequency content, and combines the remnant upper-spatial-frequency content of the aerial photographs containing desired details with the low spatial frequencies of the satellite data (or other very high altitude data) as a substitute for the discarded lower-spatial-frequency content of a mosaic of the aerial photographs. This combining procedure generates a terrain map relatively free of radiometric distortion.
Satellite cameras are normally electronic cameras, rather than film cameras, and supply data in digitized form. Satellite cameras are often of the pushbroom type, using a line array of photosensors disposed perpendicularly to the direction of satellite travel to generate scan lines, with line advance being provided for by the satellite motion around its orbital path. It is usually preferable to use satellite data only in the visible spectrum, since objects reflect disparate wavelengths differently. In fact, the satellite camera images should be gamma-corrected and color-corrected to match, as closely as is practical, the color rendition of the chain including the aerial camera and its film and further including the video camera scanning the aerial photographs.
The satellite camera image must be geometrically registered to the image from the aerial photograph or mosaic of such photographs in order to implement the substitution of lower-spatial-frequency content from the satellite camera for the discarded lower-spatial-frequency content of a mosaic of the aerial photographs. As part of the registration procedure, the distortion effects owing to satellite camera angle and to perspective are corrected for in a computer provided with the digital data from the satellite camera, and optical distortions in the camera lens may be corrected for. Then, the low spatial frequency band of the mosaic of images from the aerial camera is replaced by the low spatial frequency band of the satellite camera image, after suitable relative intensity correction.
The upper-spatial-frequency content of the mosaic of aerial photograph images not only contains desired details, but also tends to contain undesirable upper-spatial-frequency artifacts attributable to the mismatching between adjacent images. Such mismatching introduces step changes into the mosaic image that, owing to the spatial bandwidth limitations imposed by the image being described in sampled data terms, appear as ringing phenomena in the spatial-frequency domain. It is desirable to suppress these undesirable upper-spatial-frequency artifacts attributable to the mismatching between adjacent images accompanying the desired details before combining the remnant upper-spatial-frequency content of the aerial photographs with the low spatial frequencies of the satellite data (or other very high altitude data) by smoothing the transitions between overlapping images.
Techniques are available to further the suppression of these undesirable artifacts which rely on equalization of the intensities of the images in their regions of overlap using histogram methods. A thesis "COMPENSATION OF NONUNIFORM IMAGE PLANE IRRADIANCE FOR AERIAL MOSAICS" submitted by Bena L. Currin in May 1990 to the graduate faculty of Rensselaer Polytechnic Institute in partial fulfillment of the requirements for a Master of Science degree is incorporated herein by reference, particularly for its description of those equalization methods. The thesis illustrates with photographs the processing of images in order to generate a terrain map of the Malibu area, for use in a flight simulator. Currin also discloses combining the upper-spatial-frequency content of aerial photographs with the low spatial frequencies of satellite data.
Another good technique for smoothing the transitions between overlapping images is described by P. J. Burt and E. H. Adelson in their paper "A Multiresolution Spline with Application to Image Mosaics", ACM Transactions on Graphics, pp. 217-236, October 1983. This transition-smoothing technique can be used either to avoid or to augment adjusting the intensities of the images in their regions of overlap using histogram methods. The Burt and Adelson image-splining technique makes use of a localized spatial-frequency spectrum analysis technique known as the "pyramid transform" which the reader should be cognizant of to more fully understand certain aspects of the invention disclosed in this specification.
A good thumbnail description of the Burt pyramid transform is provided in the "Description of the Prior Art" portion of the "Background of the Invention" of U.S. Pat. No. 4,674,125 issued Jun. 16, 1987, to C. R. Carlson, J. A. Arbeiter and R. F. Bessler; entitled "REAL-TIME HIERARCHAL PYRAMID SIGNAL PROCESSING APPARATUS"; now re-assigned to General Electric Company and incorporated herein by reference. U.S. Pat. No. 4,661,986 issued Apr. 28, 1987, to E. H. Adelson; entitled "DEPTH-OF-FOCUS IMAGING PROCESS METHOD" and now re-assigned to General Electric Company describes how an improved-focus two-dimensional image is derived from original images of the same field-of-view taken with the same camera differently focussed. The respective Burt pyramid transforms of the original images are obtained, each providing a spectral analysis of an original image by octaves. The Burt pyramid transform of the improved-focus two-dimensional image is assembled octave by octave, choosing the corresponding octave of the original image having the highest intensity level. The improved-focus two-dimensional image is then generated from its Burt pyramid transform by performing an inverse pyramid transform procedure. Accordingly, the generation of a synthetic image by performing an inverse pyramid transform procedure on a Burt transform assembled octave by octave from the Burt pyramid transforms of other images each of the same field-of-view is generally known. A bibliography of the early literature concerning the Burt pyramid transform is found in the "Background of the Invention" of either of the U.S. Pat. Nos. 4,661,986 and 4,674,125.
Digital filtering for the spatial-frequency spectrum analysis is associated with obtaining the Burt pyramid transform in order to implement transistion smoothing. This digital filtering can also be used for separating the upper-spatial-frequency content of aerial photographs from their lower-spatial-frequency content. Digital filtering of the satellite data for obtaining its Burt pyramid transform facilitates combining the lower-spatial frequency bands of image data in that Burt transform with the higher-spatal frequency bands of image data in the Burt transform obtained from the mosaic image generated from aerial camera photographs, without incurring visible artifacts attributable to that combining.